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Abstract 


Community colleges are a large sector of postsecondary education. In 2016-2017, the United States 
had nearly 1,000 public 2-year postsecondary institutions (community colleges), serving almost nine 
million students, representing 39% of all undergraduates. The majority of entering community college 
students require developmental (or remedial) math. Success rates in the developmental math course 
sequence and college more broadly are discouragingly low. Policymakers, practitioners, and research- 
ers alike are eagerly searching for reforms to improve success rates, but there is a dearth of causal 
evidence on the effectiveness of most proposed reforms. We sought to answer the following question: 
What effect does a modularized, computer-assisted, self-paced approach to developmental math (com- 
pared with a more “traditional” direct-instruction course alternative) have on students’ likelihood of 
completing the developmental math course sequence? Findings from a randomized controlled trial 
(n=1,403) are presented. The program was well implemented; however, we did not find evidence that 
this approach was superior to the “traditional” math class. Although these results are disappointing, 
they are important because modularization and self-paced computer-assisted instruction are popular 
reforms. 
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Introduction 


In the United States, community colleges play a vital role in advancing the nation’s agenda to 
increase college degree attainment and technical skills training, as they serve a significant pro- 
portion of postsecondary education students. In 2016-2017, nine million students attended public 
2-year colleges (community colleges), representing 39% of all undergraduates. The open admis- 
sions and relatively low cost of community colleges (compared with 4-year colleges and univer- 
sities) have contributed to unprecedented access to postsecondary education. However, rates of 
successful degree completion leave much room for improvement. Nationwide, among first-time, 
full-ttme community college students in the 2013 cohort, only 25% graduated within 3 years 
(Ginder, Kelly-Reid, & Mann, 2017). 


One of the greatest challenges facing community colleges is that most entering students 
(approximately 59%) are deemed academically underprepared for college-level math and are 
referred to developmental math courses (Chen, 2016). These courses are intended to prepare 
students for college-level work; however, most students who place into remedial math courses 
never complete them, move on to introductory college-level math courses, or earn a degree 
(Bailey, Jenkins, & Leinbach, 2005; Bailey, Jeong, & Cho, 2010; Attewell, Levin, Domina, & 
Levey, 2006). 


While the low success rates of students who place into developmental education are well 
documented (Bailey, Jeong & Cho, 2010), there is a dearth of causal evidence on the effectiveness 
of strategies to improve outcomes for these students. In fact, the U.S. Department of Education’s 
What Works Clearinghouse (WWC) recently published a practice guide of strategies to help post- 
secondary students in developmental education, and after reviewing 25,697 studies, only 10 met 
the WWC evidence standards with or without reservations. Moreover, of the six practices recom- 
mended in the guide, only three had any evaluations meeting the WWC evidence standards sup- 
porting them, indicating that postsecondary administrators and policymakers desperately need 
causal evidence of strategies to improve developmental education (Bailey et al., 2016). 


The present study contributes to this literature via a rigorous evaluation of a widespread 
type of developmental education reform — the division of remedial math courses into discrete, 
single-unit modules, in which the content is delivered via self-paced, computer-assisted instruc- 
tion. The popularity of this type of reform extends from statewide policies to independent creation 
of homegrown programs at individual colleges. A 2016 survey of a nationally representative sam- 
ple of 911 2- and 4-year colleges in the United States found that 40% of institutions offered self- 
paced approaches to developmental math education, and 32% of colleges used computer-based 
learning to support underprepared students (E. Zachry Rutschow & A. K. Mayer, personal com- 
munication, February 6, 2018). In the past few years, policymakers in Virginia and North Caro- 
lina independently standardized the developmental education curricula across all the colleges 
in their respective states and modularized their developmental math courses (Kalamkarian, 
Raufman, & Edgecombe, 2015). 


At Tarrant County College (TCC) in Texas, math department faculty created ModMath, 
a modularized, computer-assisted, self-paced developmental math course sequence that was the 
subject of the present study. In this paper, we present findings from a 1,400-person randomized 
controlled trial of ModMath. The sections that follow provide context for the study, describe the 
program model and the theory of change, and detail the study’s design and results. We conclude 
with a discussion on the implications of the findings. 


Background and Study Context 


Developmental Math at Tarrant County College 


When the present study began, the developmental math sequence at TCC consisted of 
two courses: Developmental Math 1 and Intermediate Algebra. Students were placed into one of 
the two developmental math classes or into college-level math using the Texas Success Initiative 
(TSI) Assessment.' The courses were offered in a variety of course formats, including as lecture- 
based courses, computer-assisted lectures, Emporium (described later), and ModMath. 


The focus of this paper is on the effectiveness of TCC’s ModMath program. However, 
as Holland (1986) noted, “The effect of a cause is always relative to another cause” (p. 946). In 
other words, there is no singular effect of TCC’s ModMath program — it depends what it is being 
compared with. In the present study, we examined the effectiveness of TCC’s ModMath relative 
to the alternative developmental math course offerings at TCC. Table 1 provides a brief overview 
of ModMath and these alternative offerings, which is essential for understanding the service con- 
trast and thus the effects we estimate. While the control group’s experiences reflect some partic- 
ipation in all three alternatives, most control group students enrolled in lecture-based courses. In 
the Program Implementation section, we provide greater detail on participation in these alterna- 
tive offerings, as well as students’ experiences in these courses. 


The next section describes the ModMath program model and the theory of change. 


ModMath Model, Theory of Change, and Prior Research 


Program Model 


ModMath is a developmental math reform that changes the structure of the develop- 
mental math sequence and the instructional delivery of the curriculum, but not the course con- 
tent itself. ModMath’s course structure and instruction encompass four key components: 
(1) modularized courses, (2) computer-assisted instruction, (3) a diagnostic assessment, and 


‘Students may be exempt from taking the Texas Success Initiative Assessment if they obtain a certain score 
on their SAT, ACT, or Texas statewide high school test; have successfully completed a college-level math 
course; or are active or veteran members of the military. 
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Table 1. Features of Developmental Math Course Offerings 
at Tarrant County College 


Course Type Sequence Structure Instructional Delivery 


ModMath * Six 5-week modules * Computer-assisted instruction 

¢ Each mod is 1 credit * Self-paced 
* Support from instructor and aide 
* Acceleration possible 
Lecture-Based * Two 16-week courses * Instructor-led lectures 

* Each course is 3 credits * Instructor sets pace 


Computer-Assisted * Two 16-week courses * Instructor-led lectures and computer-assisted 
Lecture * Each course is 3 credits instruction (varies by instructor) 
* Instructor sets pace 
Emporium * Two 16-week courses * Computer-assisted instruction 
* Each course is 3 credits * Self-paced 
* Acceleration possible 


(4) on-demand, personalized assistance. Table 2 depicts the ModMath theory of change or logic 
model. The first two columns describe each component of the program and associated practices. 
The last two columns list intended student outcomes, and the middle column explains the mech- 
anisms through which each set of practices was hypothesized to improve outcomes for Mod- 
Math students. The following sections detail the program components and the theory of change 
and summarize current research on the efficacy of this type of developmental math reform. 


Modular Courses: The core component of ModMath is a structural change that divides 
each of the two semester-long developmental math courses into three 5-week one-credit modules 
or “mods” (Mods 1-6). All six modules are offered in any given ModMath course section. Stu- 
dents typically enroll in three modules — the equivalent of one traditional developmental math 
course — each semester. 


Computer-Assisted Instruction: ModMath’s course content is delivered via Pearson’s 
instructional software program called MyMathLab. Students meet in computer classrooms at reg- 
ularly scheduled times and work independently through the course material using instructional 
videos, PowerPoint slides, or an online or hardcopy version of the textbook. Because instruction 
is self-paced, students may accelerate and complete more than three modules in a given semester. 


Diagnostic Assessment: To place students into one of the six computer-assisted mod- 
ules, ModMath supplements the college’s standard math placement test — the TSI Assessment 
— with an additional placement exam, Pearson’s MyMathTest.2 MyMathTest is intended to be 
more fine-tuned, or precise, than TCC’s standard placement exam to place students in a module 


>The exam, which is developed using Pearson’s MyMathTest platform, allows faculty to create a custom- 
ized placement exam by selecting questions from the software’s test bank of problems that align with TCC’s 
developmental math curriculum. 


Table 2. A logic model for ModMath: components, practices, mechanisms, and outcomes. 


Component Key Practices and Features Mechanisms Student Outcomes 
Diagnostic . Fine-tuned for module placement . Accurate placement resulting in close 
Racgeament * Aligned with course content alignment of content with students’ prior 
(MyMathTest) math knowledge Long-Term 
Academic Progress 
+ Each semester-long course divided * Students earn one credit for each mod Short-Term 
into three 5-week modules passed, resulting in sense of progress Academic Progress 
¢ Each module worth 1 developmental + Students who fail or stop attending a mod 
Modularized math credit can repeat in the next 5-week session 
Courses ¢ Modules align with the standard rather than wait until next semester to 
curriculum repeat whole course, resulting in increased 
persistence 
: Bono the College-level credits 
* 100 percent of class time in computer = Allows for self-paced learning developmental math earned : 
lab ¢ Allows for completing up to 6 modules in a sequence completed Total credits earned 
¢ Various content delivery methods for single semester * College-level math Persistence 
Computer- instruction (video, presentation, text- | * Students move on only after demonstrating course completion Degree/certificate 
Based book) mastery of material Grade sammed it attainment 
Instruction * Mastery learning * Variety of content delivery methods and suibiects-otharthan * Transfer rates to 
(MyMathLab) —.« Extra support available via software frequent assessments facilitate mastery aan 4-year colleges 
* Each class staffed with an instructor * Increases the amount of one-on-one 
and aide instructor—student interactions, allowing 
: * Instructor and aide circulate during more academic and emotional support 
Personalized, class, providing one-on-one 
On-Demand assistance 
Assistance 


Source: Adapted from Gardenhire, A., Diamond, J., Headlam, C., & Weiss, M. J. (2016). Note. The study looks at short-term academic progress (three semesters 
after random assignment). 


that closely matches their demonstrated math knowledge and skill deficits. Students are then 
placed in one of the six modules, depending on their skill needs as determined by MyMathTest 
results, and must complete the remaining set of modules to complete developmental math. 


On-Demand, Personalized Assistance: ModMath is designed so that instructors serve 
as facilitators who provide individualized support to students, as opposed to whole-group instruc- 
tion. Each class, which typically enrolls 24 students, is staffed by an instructor and an instructional 
aide. Compared with traditional lecture-based courses, the highly personalized structure of Mod- 
Math provides more opportunities for instructional staff to provide one-on-one academic and 
emotional support to students. 


Theory of Change 


ModMath is theorized to improve student outcomes by addressing common challenges 
to developmental math instruction. A major pedagogical challenge in developmental math is that 
the courses typically serve students with a wide range of academic abilities, learning styles, and 
personal needs. Unless an instructor can accommodate these differences through differentiated 
instruction and other methods, such as supplemental academic support, students may fall behind, 
disengage, or fail (Tomlinson & Kalbfleisch, 1998). Several ModMath components allow for 
greater differentiation than is feasible in traditional lecture-based classrooms. 


First, the modular structure, coupled with the fine-tuned diagnostic exam, is theorized to 
allow for more precise placement than the standard assessment used for traditional math courses. 
This granular type of assessment should allow students to enter a modular sequence at a level that 
is closely aligned with their prior knowledge, so that they focus only on topics in which they need 
remediation (Bickerstaff, Fay, & Trimble, 2016; Bracco et al., 2015). Second, ModMath tailors 
instruction to each student by combining computer-based learning via the instructional software 
package with on-demand, personalized assistance from the instructor and aide. This “hybrid” 
learning environment has been linked to improved student outcomes (Chekour, 2017; Means et 
al., 2009), although the evidence is mixed (Xu & Jaggars, 2011). Instructional software is thought 
to accommodate students with various learning styles and abilities by providing diverse instruc- 
tional materials and allowing for self-paced instruction. Both features are intended to increase 
student engagement and performance (Goldschmid & Goldschmid, 1973; Subban, 2006). Fur- 
thermore, self-paced instruction should allow some students to accelerate and complete more than 
the standard three modules per semester. 


In general, instructional software, such as MyMathLab, is theorized to facilitate optimal 
learning by placing students in Vygotsky’s (1978) zone of proximal development — the devel- 
opmental level where instructional content is just beyond students’ current knowledge and stu- 
dents are capable of learning new material with assistance from a more knowledgeable tutor. In 
ModMath, this tutoring can come from the software, which includes several help features to guide 
students through new math problems, or from the instructor, who can fill in conceptual gaps or 
provide alternative methods of problem-solving to those provided by the software. Computer- 
assisted instruction is also theorized to facilitate mastery learning because it allows students to 


take frequent assessments, receive formative feedback on their learning, and progress only when 
they have mastered the material. This process is thought to improve student performance in de- 
velopmental and in more advanced math courses (Bishop, 2010; Epper & Baker, 2009; Twigg, 
2005). 


Another common challenge to developmental math instruction is that many students of- 
ten have a history of underperformance in math, suffer from math anxiety, or lack confidence in 
their mathematical abilities (Dwinell & Higbee, 1989; Taylor, 2006). By staffing each course 
with an instructor and an aide who provide individualized assistance, ModMath increases instruc- 
tor-student interactions and allows opportunities for instructional staff to provide not only aca- 
demic but also emotional support to students. This support may build students’ confidence and 
improve their academic progress in math, as greater connections to staff can increase students’ 
feelings of belonging and facilitate persistence (Tinto, 1999). 


Finally, another challenge to developmental math instruction, especially in community 
colleges, is that many students stop attending classes or drop out for a variety of reasons. In tra- 
ditional courses, students who drop out mid-semester do not receive any credit for already com- 
pleted work and must restart the course from the beginning. The modularized structure of Mod- 
Math allows students to leave and return without losing as much ground. Furthermore, students 
using ModMath earn credits incrementally, which may give them a sense of accomplishment as 
they experience “small wins” (Weick, 1984) toward the goal of completing the entire develop- 
mental math sequence. 


Effectiveness of Modularized, Computer-Assisted Developmental 
Math Programs 


There is limited causal evidence on the efficacy of modularized, computer-assisted de- 
velopmental math courses like ModMath. Much of the research relies on descriptive studies that 
examine academic outcomes, and the findings from these studies are mixed. For example, 
Squires, Faulkner, & Hite (2009) found that computer-assisted modules resulted in higher course 
pass rates in developmental math and subsequent college-level courses. On the other hand, 
Kalamkarian et al. (2015) studied North Carolina’s and Virginia’s modularized developmental 
math programs and concluded that pacing and attrition were major issues in self-paced, computer- 
assisted modules, as a significant portion of students did not complete the expected number of 
modules for the given semester. Ariovich & Walker (2014) found mixed results within a single 
study: students who opted to take modularized, computer-assisted developmental math performed 
worse in lower-level courses but better in higher-level courses, compared with students in lecture- 
based developmental math. 


The mixed results of modularized, computer-assisted, self-paced developmental math 

courses may relate to variations in program implementation and institutional context. Fay (2017) 

concluded that high school students in Tennessee experienced higher pass rates than community 

college students using the same computer-assisted modularized developmental math program be- 

cause the high schools had more structured classroom environments, such as rigorous attendance 

policies and frequent class meetings, than the community colleges, which were characterized by 
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more flexibility and student autonomy. Bickerstaff et al. (2016) found that stand-alone modules 
(in which all students are in the same module) allow for instructional flexibility (as they can be 
taught via lecture or instructional software) but create additional exit points at which students may 
fail to reenroll in the next module. On the other hand, computer-assisted courses that offer several 
modules in one course section reduce the number of exit points but may slow student progression. 


Importantly, the findings from all the aforementioned studies rely on descriptive research 
that does not use a comparison group. At least one quasi-experimental study has been conducted. 
Following a statewide redesign of remedial education in Tennessee, some community colleges 
divided their developmental math sequence into computer-assisted modules. Boatman (2012) 
evaluated two programs of this type and found similarly mixed results. 


The inconclusive literature on modularized, computer-assisted courses highlights the 
need for a causal, experimental evaluation of the approach. The present study aims to begin to fill 
this gap. 


Evaluation 


Research Questions 


The primary goal of the evaluation was to answer the following question: 


e What effect does the opportunity to enroll in ModMath (compared with the 
opportunity to enroll in the college’s “traditional” math courses) have on stu- 
dents’ likelihood of completing the developmental math course sequence? 


We were also interested in any positive spillover effects or negative side effects on pro- 
gress outside of math that may be caused by ModMath. 


In addition to this overarching goal, we sought to understand several questions related to 
the implementation of ModMath and the “traditional” math courses with which ModMath is com- 
pared. They included: 


e To what degree were ModMath services and activities implemented as 
planned (i.e., how strong was their fidelity to the program model)? 


e To what degree were the services experienced by program group students dif- 
ferent from those experienced by control group students (i.e., to what degree 
was there a contrast between the program and control conditions)? 


>This question and the implied primary outcome of interest were specified in an internal (i.e., not published) 
analysis plan written before to conducting analyses. The main outcome (confirmatory) of interest was having 
successfully completed the developmental math course sequence (for more on what is meant by “confirmatory,” 
see Schochet, 2008). 


The Study College: Tarrant County Community College, Texas 


The study took place at Tarrant County College (TCC) District. With six campuses and 
an annual enrollment of about 50,000 students, TCC is one of the largest community college 
systems in Texas. This study took place at the Northeast Campus, located in Hurst, a suburb 
between Fort Worth and Dallas. The campus offers a range of associate degree programs that 
prepare students to enter professional careers or transfer to 4-year institutions. At the launch of 
this study, the Northeast Campus served approximately 15,000 students, and about 1,200 students 
were enrolled in developmental math education at the college (TCC Office of Institutional Re- 
search, Planning, and Effectiveness, as cited in Gardenhire, Diamond, Headlam, & Weiss, 2016). 


Design 


We used a random assignment research design to estimate the effect of the ModMath 
program compared with a “business-as-usual” control condition at the college, generally a more 
traditional, lecture-based course. Included in the evaluation were eligible students (those in need 
of developmental mathematics, based on placement test scores) who (a) were willing to partici- 
pate in the ModMath program, (b) filled out a baseline survey, and (c) signed an informed con- 
sent.4 After completing the baseline survey and informed consent, students were randomly as- 
signed, through a computer algorithm controlled by the research team, to either the program or 
the control group. Program group members had the opportunity to participate in ModMath. Con- 
trol group members had the opportunity to participate in any of the college’s other developmental 
math course offerings and any support services, just not ModMath. Students had roughly a 60% 
chance of being assigned to the program group. Four cohorts of students were randomly assigned, 
one prior to the spring and fall semesters in 2014 and 2015. In total, 1,408 students were randomly 
assigned. Five students were not included in any analyses (two in the program group, three in the 
control group) because they withdrew from the study or their consent form was not recovered, 
leaving an analytical sample of 1,403 students.° 


Data Sources 


Data for this mixed methods evaluation came from qualitative and quantitative sources. 
These data sources are briefly described below; for more details, see Gardenhire et al. (2016). 


Qualitative data were collected from focus groups with five groups of students (ranging 
in size from four to 20 participants), one group of ModMath instructors and one group of tradi- 
tional developmental math instructors (17 participants, total), and one group of academic ad- 
visers (seven advisers). Interviews with 12 Tarrant County College staff members involved in 
the ModMath program were conducted. Finally, the research team informally observed Mod- 
Math classrooms 10 times and control group classrooms five times. The qualitative data were 


4For more details on the recruitment process, see Gardenhire, Diamond, Headlam, & Weiss (2016). 
>The overall attrition rate is 0.36% and the rate of differential attrition (the difference between program 
group attrition and control group attrition) is 0.28 percentage point. 
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used primarily to understand the program implementation and the experiences of program and 
control group members. 


Quantitative data included a baseline survey (taken just before to random assignment, 
when students first enrolled in the study), TSI placement test records (used to determine develop- 
mental math course placement for most control group students), MyMathTest placement scores 
(used to determine module level for most program group students), student transcript records, and 
a student survey conducted during students’ first semester after random assignment. ° 


Student Characteristics 


As described above, upon joining the study, students completed a baseline survey cover- 
ing information about their demographic characteristics, family and educational backgrounds, 
and experiences with math. As Table 3 shows, about 64% of students in the study were female. 
The study sample was 46% white, 28% Hispanic, and 20% black.” Many students in the study 
had characteristics associated with a low likelihood of academic success (Engle, 2007). For in- 
stance, a third of the students were the first in their families to attend college. Only 43% planned 
to enroll in school full time during the first study semester. Nearly three-quarters said that they 
planned to work during the upcoming semester, and about 44% planned to work full time. In 
addition, more than half of students reported that they had failed a math class in the past.® 


Outcomes 


The primary outcome of interest was completion of the developmental math sequence. 
To provide context when interpreting ModMath’s effect on completing developmental math, sev- 
eral additional outcomes were examined. All outcomes are described briefly: 


Participation: Two indicators of participation were considered: 


e Enrolled in college. Enrollment was defined as of the add-drop deadline in a 
given semester. 


e Enrolled in any math class. Enrollment in any math class was defined based 
on enrollment in a module or a math course in a given semester. 


Developmental Math Progress/Completion: Several measures of developmental math 
progress and/or completion were examined: 


The student survey had a 73% response rate, including program and control students. 
’This sample reflects a gender and racial composition like that of Tarrant’s overall student body. 


8See Appendix A.1 in Gardenhire et al. (2016) for a comprehensive list of data reported by students on the 
baseline form, as well as a comparison of program and control group students on these measured characteristics. 
The two groups are strikingly similar on all characteristics. 
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Table 3. Characteristics of students in the study. 


Full sample 
Characteristic (%) (N = 1,389) 
Female 64.3 
Age 
18 and under 22.0 
19-24 38.4 
25-34 21.7 
35-44 10.4 
45 and over 7.5 
Race/ethnicity 
Hispanic 28.3 
White 45.7 
Black 19.6 
Other 6.5 
Completed 12th grade 87.3 
First person in family to attend college 33.7 
Planned enrollment this semester 
Less than part time (fewer than 6 credits) 17.8 
Part time (6-12 credits) 39.0 
Full time (12 credits or more) 43.2 
Planning to work this semester 
No 18.9 
Yes, part time (less than 30 hours a week) 31.0 
Yes, full time (30 hours a week or more) 43.6 
Missing 6.6 
Failed a math class in the past 53.5 
Missing 6.7 
TSI Math Placement 
College-ready or exempt 6.8 
Placed one level below college-ready 9.1 
Placed more than one level below college-ready 52.0 
Math placement information is unknown or missing 32.1 


Note. N = 1,389. Rounding may cause slight discrepancies in sums and differences. Missing 
shows the percentage of survey respondents who did not answer the question. Missing values 
are reported only for items with more than 5% missing. Calculations were made using data from 
the baseline survey of TCC students and TCC placement test data. 

Respondents who said they were Hispanic and chose a race are included only in the 
Hispanic category. Respondents who said they were not Hispanic and chose more than one 
race are included in the Other category. The Other category also includes respondents who 
chose Asian, American Indian, or Pacific Islander. 

Includes students who were found to be ready for college, who received waivers from testing 
requirements, who were exempt from testing requirements, or who had previously completed 
testing requirements. 

“Includes students who were not included in the TCC placement test data. 
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e Average percentage of developmental math sequence completed. One indi- 
cator of progress was the percentage of the developmental course sequence 
completed.’ Because students in ModMath can achieve smaller increments 
of success than students in traditional courses, this measure may favor the 


program group. 


e Earned at least 1 developmental math credit. Another indicator that a student 
made any progress is whether he or she passed at least one module (one credit) 
or one course (three credits) since random assignment. Since students in Mod- 
Math can complete a one-credit module and students in traditional courses 
must complete an all-or-nothing three-credit course, this measure of progress 
favors the program group. 


e Completed first half of developmental sequence. A final indicator of progress 
was whether students surpassed the halfway milestone in the developmental 
sequence — Mod 3 or Math 0361 (the lower-level developmental math 
course). This may be the fairest progress milestone comparison, since Mod- 
Math and the traditional course sequence have a similarly defined halfway 
point. 


e Completed developmental sequence (college-ready). The primary outcome of 
interest was completion of the developmental math sequence, which is defined 
as having passed Mod 6 or developmental Math 0362 since random assign- 
ment. Students who achieve this milestone are considered “college-ready” in 
math and may proceed to college-level course work. 


Results 


Program Implementation and Contrast Between Program 
and Control Conditions 


Overall, we found that the program components were implemented with strong fidelity 
to the model and that the experience of the ModMath program group was different in anticipated 
ways from the experience of the control group. 


ModMath courses were offered as discrete 5-week, one-credit modules, as planned, and 
more than 80% of students in the program group enrolled in a ModMath course during their first 
semester after random assignment. Classes were held in computer labs, and more than 70% of 
students randomly assigned to ModMath took the MyMathTest diagnostic exam to determine 


*For example, a program group student who completed Mod | but not Mod 2 would be considered to have 
completed one-sixth of the developmental sequence, or about 17%. A student who completed Mod 2 but not 
Mod 3 would be considered to have completed two-sixths of the sequence, or about 33%. For control group 
students, a student who completed the developmental course Math 0361 (lower-level math) would have com- 
pleted half of the sequence, or 50%, while a student who completed the developmental course Math 0362 would 
have fully completed the sequence. 
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their starting module. Instruction was delivered primarily via the MyMathLab software, and each 
class was staffed with an instructor and aide, who provided individualized assistance. 


In comparison, the control group enrolled in semester-long, three-credit courses. There 
was variation in the types of courses control group students enrolled in. Approximately 57% of 
students enrolled in traditional lecture-based courses, and approximately 20% of students enrolled 
in one of the non-modularized alternative developmental math offerings, either computer-assisted 
lecture or Emporium. This is unsurprising, as control group students could enroll in any develop- 
mental math course type, except ModMath. Table 4 shows the percentage of students who en- 
rolled in each course type. (For simplicity, we have classified computer-assisted lecture and Em- 
porium as non-modularized computer-assisted courses.) Regardless of the type of course in which 
they were enrolled, all control group students had access to the MyMathLab software package, 
as the software was required for homework. In addition, students in non-modularized computer- 
assisted courses used MyMathLab for in-class instruction to varying degrees. 


Table 4. First program semester math course type enrollment. 


Program group Control group Estimated Standard 
Outcome (%) (N = 826) (N = 577) effect error 
First semester 
ModMath 83.1 0.2 82.9*** 1.6 
Lecture based 2.1 56.9 -54.8*** 1.8 
Non-modularized computer assisted 0.5 20.4 -19.9*** 1.4 
Other? 1.1 3.0 -1.9*** 0.7 
No math course 13.3 19.6 -6.3*** 2.0 


Note. N = 1,403. Estimates are adjusted by cohort. Rounding may cause slight discrepancies in sums 
and differences. 

"Includes math courses where the section type is unknown. Calculations made using transcript data 
from TCC. 

**p = .01. “p= .05. *p=.10. 


Even though students in both groups had similar access to the instructional software, 
classroom observation and student survey data confirmed a significant contrast between the class- 
room experience of ModMath program and control group students. ModMath instructors focused 
on providing one-on-one assistance during class. As shown in Table 5, 68% of ModMath students 
reported that their instructors spent a considerable amount, or most, of their time working indi- 
vidually with students during class, compared with only 32% of control group students. In con- 
trast, the majority of non-ModMath instructors focused class time on whole-group instruction. 
Seventy-nine percent of control group students reported that their instructors spent a considerable 
amount, or most, of their time lecturing the class, compared with only 23% of ModMath students. 


Accordingly, there was a significant difference in how students spent their time in math 
class. ModMath students worked individually on the instructional software. Table 5 shows that 
approximately 80% of ModMath students reported spending most of their time in class using 
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Table 5. Student survey results: instruction and assistance. 


Sample Program Control Standard 
Response (%) size group group _ Difference error 
In your most recent math class, the instructor spent a 
considerable amount or most of the class period 
Lecturing the class 993 23.3 79.3 -56.0*** 2.7 
Working with small groups of students 992 22.8 25.0 -2.2 2.8 
Working with students individually 992 67.6 31.7 35.9*** 3.0 
Giving announcements not related to math 989 12.4 14.4 -1.9 2.2 
In your most recent math class, the students spent a 
considerable amount or most of the class period 
Working alone on math exercises 992 80.5 53.6 26.8*** 2.9 
Working in small groups on math exercises 991 7.3 12.9 -5.6*** 1.9 
Working as a class on math exercises 990 15.1 60.5 -45.4*** 2.7 
Chatting, texting, or on personal business 991 4.1 6.3 -2.1 1.4 
Using computers, calculators, or technology 993 80.5 57.8 22.6*** 2.9 
Having problems with technology 993 2.5 5.9 -3.4*** 1.2 


Note. Total survey respondents N = 1,012. Total program group respondents N = 620. Total control group 


respondents N = 392. Estimates are adjusted by cohort. Rounding may cause slight discrepancies in sums and 


differences. Distributions may not add to 100% because categories are not mutually exclusive. Calculations 
made using data from survey of TCC students. 
*™*o9 = .01. **p = .05. *p = .10. 


computers, calculators, or technology and working alone on math exercises, while close to 60% 
of the control group reported similar experiences. This was expected, given that the question 
asked whether students used computers, calculators, or technology, and students in non-Mod- 
Math classes had time allotted for individual and small group exercises. Generally, control group 
students reported more whole-group instruction. Sixty-one percent of the control group report 
spending a considerable amount, or most, of their time working as a class on math exercises, 
compared with only 15% of the program group. 


Taken together, these data highlight that ModMath offered students a more individual- 
ized class experience in which students spent significant time working independently on math 
exercises on their computers or being assisted by the instructor individually. In contrast, non- 
ModMath instructors spent a considerable amount of time lecturing the class, and students 
worked as a class on math exercises. 


Evidence for ModMath’s Theory of Change 


We examined whether the expected benefits of certain program components (as outlined 
in the theory of change) were realized in practice. 


Fine-tuned assessment and module placement. Modules in conjunction with the fine- 
tuned diagnostic exam did not allow the majority of ModMath students to bypass already mas- 
tered material and start in a higher module. Approximately 84% of students placed at the begin- 
ning of the math course sequence under both testing approaches — Mod 1 under MyMathTest or 
Developmental Math 1 under other placement tests. Since these students likely needed remedia- 
tion in all the content covered by the developmental math curriculum, the opportunity to place 
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into modules using MyMathTest did not make much of a difference.'° Relevant context when 
considering this finding is that when ModMath first began, prior to the present study, TCC offered 
three developmental math courses and nine ModMath modules. Just before the start of the study, 
the lowest developmental math course was cut from the sequence due to a series of statewide 
policy changes, and the number of modules was reduced from nine to six. This change may have 
resulted in a greater number of students beginning the developmental math sequence in the lowest 
course or module, limiting the potential benefit of the diagnostic assessment. 


Modularization and credit retention: As hypothesized (and shown later), the modular 
structure resulted in greater credit accumulation, as the program group earned more develop- 
mental math credits each semester compared with the control group, because they were able to 
earn one credit at a time (see Program Effects on Academic Outcomes section below). Focus 
group data suggest that this resulted in a greater sense of accomplishment. For example, one 
participant commented: “I’ve taken many, many remedial classes and haven’t gotten anywhere 
because after a while I realized that I didn’t have the core that I needed to move on. I’m almost 
finished with my college career here at TCC, and the only thing that’s holding me back is the 
math requirements. So I was really upset that I had to start at the bottom [mod], but I will say I 
passed my first mod with an A, which I’ve never done that, so something must be working.” 
This quote illustrates a common sentiment by focus group participants who received transcript 
credit mid-semester for passing a module. By facilitating these small wins, ModMath is theo- 
rized to result in improved developmental math completion. We examined whether this oc- 
curred in practice, as described below. 


Content mastery. Because there was not a common final exam for program and control 
group students, the current study could not directly assess whether ModMath resulted in improved 
content mastery. It is important to note that ModMath students could attempt their exams without 
working through the module contents; however, they could not progress to the next module with- 
out passing the exams for their current module. 


Self-paced learning and acceleration. Computer-delivered modules are theorized to 
improve student outcomes because they allow students to work at their own pace and poten- 
tially complete more than three modules in one semester. Our analyses revealed that accelera- 
tion was possible but not prevalent. Focus group participants often referred to the self-paced 
nature of ModMath as being beneficial in allowing them to slow down when learning the ma- 
terial, as opposed to accelerating and completing the sequence faster. For example, one student 
commented that with the video lessons, “you get to stop the teacher and keep going over and 
over [the material] until you get it, and you won’t move on, and you won’t get left behind 
because the teacher or class is moving on.” This finding was confirmed by transcript data, 
which revealed that more ModMath students repeated modules, as opposed to accelerating and 
completing more than three modules in one semester (24% vs. 1%, respectively). This occurred 
even though instructors provided students with a pacing calendar that detailed the amount of 


'‘OMyMathTest did appear to alter placement for 16% of students: approximately 12% of students placed 
higher, and approximately 4% of students placed lower than under the college’s standard placement exams. 
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work they needed to accomplish each day to complete the module in the allotted time frame 
and the full course by the end of the semester. 


Although ModMath did not lead to student acceleration, the self-paced nature of the pro- 
gram may have resulted in students feeling that the difficulty of their course was appropriate. 
Approximately 71% of ModMath students reported that the level of difficulty of their math class 
was just right, compared with 51% percent of non-ModMath students. 


Individualized instruction and student support. Based on interview and focus group data, 
ModMath increased one-on-one interactions between instructors and students, creating more op- 
portunities for instructors to provide academic and emotional support to struggling students. Mod- 
Math focus group participants frequently compared their support experience in ModMath with 
their previous experience in lecture-based courses. For example, one student commented: “It’s so 
much easier than lectures because I don’t feel pressure to just hurry up and just understand it...I 
don’t really like asking questions in front of a big group of people because I’m scared I’m gonna 
ask a stupid question.” Most focus group participants felt that ModMath had increased their level 
of academic and personal support. 


In general, ModMath services were largely implemented as planned and the program 
provided a notably different developmental math experience for students, compared with TCC’s 
traditional lecture-based courses. There is evidence that some elements of the theory of change 
were realized in practice, while others were not. The next section discusses the program’s impact 
on students’ academic outcomes. 


Program Effects on Academic Outcomes 


This section presents estimates of the effect of ModMath on students’ academic outcomes 
achieved over the course of three semesters after random assignment. Overall, ModMath was no 
more or less effective than traditional math. After three semesters, 23% of program group mem- 
bers had completed the developmental math sequence, the primary outcome of interest. Similarly, 
22% of the control group had achieved the same milestone. On average, program group members 
completed a higher proportion of the six-credit developmental math sequence than did control 
group members. This was largely a result of the structural shift that allowed ModMath students 
to earn one credit for completing each of the six modules, whereas control group students earned 
credit only for completing the entire three-credit course. More details are provided below. 


Enrollment, Progress, and Completion of Developmental Math 


Table 6 presents information on students’ enrollment at college and progress through and 
completion of the developmental math sequence. Table 7 presents academic progress in non-math 
courses to examine spillover effects. Findings are presented for the first three semesters after stu- 
dents were randomly assigned — meaning that, regardless of cohort of entry, we present findings 
for each student’s first, second, and third semesters after entering the study. Recall that most 
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Table 6. Developmental math progress. 


Program group Control group [Estimated Standard 


Outcome (% N = 826 N=5/77 effect error 
First semester 

Enrolled in college 93.1 90.2 2.9% 1.5 
Enrolled in any math class (course or mod) 86.7 80.6 6:1" 2.0 


Average percent of developmental 
math sequence completed 26.4 17.5 8.9*** 1.4 


Developmental math sequence 
progress since RA 
Earned at least 1 developmental math 


credit or higher 69.5 30.0 39.5*** 2.5 

Completed first half of developmental sequence 27.1 29.8 -2.7 2.4 

Completed developmental sequence 

(college-ready) 2.2 5.1 -2.9*** 1.0 
Completed a college-level math course 0.4 0.3 0.0 0.3 


Second Semester 
Enrolled in college 68.7 61.8 6.9*** 2.6 


Enrolled in any math class (course or mod) 51.4 40.3 11.2*** 2.7 


Average percent of developmental 
math sequence completed 37.9 27.1 10.9*** 2.0 


Developmental math sequence 
progress since RA 
Earned at least 1 developmental math 


credit or higher 72.4 38.4 34.0*** 2.5 

Completed first half of developmental sequence 40.6 38.2 2.4 2.7 

Completed developmental sequence 

(college-ready) 14.7 15.7 -1.0 1.9 
Completed a college-level math course 2.6 2.9 -0.3 0.9 


Third Semester 
Enrolled in college 51.9 48.8 3.1 2.7 


Enrolled in any math class (course or mod) 35.3 29.9 5.5** 2.6 


Average percent of developmental 
math sequence completed 43.2 32.5 10.7*** 2.1 


Developmental math sequence 
progress since RA 
Earned at least 1 developmental math 


credit or higher 74.4 43.2 31.2*** 2.5 

Completed first half of developmental sequence 45.7 42.8 2.9 2.7 

Completed developmental sequence 

(college-ready) 22.5 22.0 0.4 2.3 
Completed a college-level math course 6.9 9.2 -2.3 1.5 


Note. N = 1,403. RA = random assignment. Estimates are adjusted by cohort. Rounding may cause slight 
discrepancies in sums and differences. The fall 2016 cohort has data only for part of the third semester. Calculations 
made using transcript data from TCC. 

**o = .01. **p=.05. *p=.10. 
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Table 7. Cumulative non-math progress. 


Program group Control group Estimated Standard 


Outcome (N = 826) (N = 577) effect error 
One semester 

Non-math credits attempted 6.49 6.56 -0.07 0.22 
Non-math credits earned 4.82 4.73 0.10 0.23 
Two semesters 

Non-math credits attempted 11.76 11.67 0.10 0.43 
Non-math credits earned 8.64 8.49 0.14 0.42 
Three semesters 

Non-math credits attempted 15.49 15.16 0.33 0.61 
Non-math credits earned 11.44 11.14 0.30 0.58 


Note. N = 1,403. Estimates are adjusted by cohort. Rounding may cause slight discrepancies in 
sums and differences. The fall 2016 cohort has data only for part of the third semester. Calculations 
made using transcript data from TCC. 

90 = .01. **p = .05. *p = .10. 


program group members needed to pass all six modules and most control group members needed 
to pass two developmental math courses to complete the developmental math sequence; thus, 
three semesters provide a reasonable amount of time for students to achieve the goal of complet- 
ing the developmental math sequence, although many students may take longer. Findings are 
presented semester by semester. 


First semester. Immediately after random assignment, 93.1% of the program group and 
90.2% of the control group enrolled at Tarrant County College. The difference, 2.9 percentage 
points, represents ModMath’s estimated effect on getting students to enroll in college.!! This may 
reflect program group students’ preference for having been given the opportunity to participate 
in ModMath or control group students’ disappointment at not being offered the opportunity to 
participate in ModMath. Alternatively, it may be the case that ModMath staff were more likely 
to conduct personal outreach to program group students to ensure they enrolled than were non- 
ModMath staff for control group students. 


In addition to this small positive effect on enrolling in college, students offered Mod- 
Math were 6.1 percentage points more likely to enroll in a math class (a module or a course), 
than were their control group counterparts. In other words, assignment to ModMath caused an 
estimated 50 additional students to enroll in math (out of the 826 program group students). ! 
This positive effect on attempting any math credits is intriguing, since, as noted by Bailey, 
Jeong, & Cho (2010), “More students exit their developmental sequences because they did not 


‘1 All analyses presented are intention-to-treat. The percentages represent least squares means, and the esti- 
mated effect was calculated using a linear regression model. For ease of exposition we refer to the effect of 
ModMath, although technically it is the effect of the opportunity to participate in ModMath. Similarly, we refer 
to program and control group outcome levels, although we are presenting regression-adjusted least squares 
means. See Appendix A for more details. 


Calculated as 826 x 0.061 = 50. 
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enroll in the first or a subsequent course than because they failed or withdrew from a course in 
which they were enrolled.” 


Notably, to be eligible for the study, students had to be willing to participate in ModMath, 
and the only way to get into ModMath was to enroll in the study. Students who preferred the 
traditional math course could simply enroll in it, without participating in the study. Thus, Mod- 
Math’s positive effect on getting students to take a math class may apply only to the types of 
students in the evaluation and may not generalize to all students in the college who required de- 
velopmental math, especially those who preferred the traditional course. Nonetheless, the offer 
of ModMath encouraged some additional students to at least attempt a developmental math class, 
which is a strong starting point.!> However, the goal of ModMath is to help students progress 
through and ultimately complete the developmental math sequence. We turned to progress and 
completion next. 


Program group students were much more likely to earn at least one developmental math 
credit (by passing a module or a course) than were their control group counterparts — an indica- 
tion of at least some degree of progress. Nearly 70% of program group students completed at least 
one class (typically at least one 5-week, one-credit module), compared with only 30% of control 
group students (typically a semester-long, three-credit course). This 39.5 percentage point in- 
crease in earning at least one math credit occurred largely because ModMath students had the 
opportunity to pass smaller portions of the developmental math sequence, 5 weeks at a time, one 
credit at a time. In contrast, students enrolled in the traditional courses were in an all-or-nothing 
situation — they had to pass an entire 16-week course to earn three credits. This structural differ- 
ence led to a substantial difference in the number of students who were able to make some formal 
progress in the developmental sequence by accumulating one or more developmental math credits 
during the first semester. 


Relatedly, on average, program group students completed a higher percentage of the de- 
velopmental math sequence than did control group students. By the end of the first semester, 
program group students had completed 26.4% of the developmental math sequence, whereas con- 
trol group students had completed 17.5% of the sequence — for an estimated effect of 8.9 per- 
centage points. The positive effect on the percentage of the developmental math sequence com- 
pleted is also due, in large part, to students earning credit for completing one or two modules in a 
semester, rather than having to complete an entire three-credit course. 


ModMath’s positive effect on completing at least one math credit and on the percentage 
of the developmental math sequence completed is encouraging. However, this success is tem- 
pered by the fact that only 27.1% percent of the program group completed the first half of the 
developmental sequence (Mod 3) by the end of the first semester. Students in the traditional math 
sequence had a similar likelihood of success — 29.8 percent of the control group made it through 
the halfway point (Math 0361) in one semester. The negative 2.7-percentage-point estimated 


Like the small effect on initial enrollment in college, the reason for the positive effect on attempting a 
developmental math course is not certain. It may have been due to the appeal of the program, but it also may 
have been an artifact of the experiment. 
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effect is not statistically significant, but it is discouraging. Moreover, there is evidence that Mod- 
Math lowered the proportion of students who completed the developmental sequence (and were 
thus college-ready) during their first semester. Only 2.2% of program group students accom- 
plished this goal, compared with 5.1% of control group students, for an estimated effect of nega- 
tive 2.9 percentage points, representing around 24 people. 


Finally, Table 7 shows that ModMath is neither helping or harming students in their non- 
math courses. This remains the same throughout the follow-up period. 


In sum, after one semester the effects of ModMath were mixed. The program helped 
students make greater progress, on average, but program group students were no more likely than 
control group students to reach the halfway milestone, and they were slightly less likely to com- 
plete the full developmental math sequence. Does the “early win” of earning at least one math 
credit, achieved by nearly 70 percent of the program group, translate into larger gains in future 
semesters? We turned to this question next. 


Second semester. Following their first semester of ModMath, 68.7% of the program 
group were still enrolled at Tarrant County College (with the option to continue taking Mod- 
Math). While this level of dropout (or stop out) is disappointing, it is an estimated 6.9 percentage 
point improvement compared with what would have occurred had program group students not 
been offered ModMath — a promising finding. Much like the first-semester enrollment effect, 
this finding may have to do with program group students’ positive experiences in ModMath or 
the control group’s disappointment at not being offered ModMath. In this case, the effect estimate 
is positive and highly statistically significant (p = .008), so it is unlikely a chance finding. 


Regarding math progress and completion, during the second semester, program and 
control group students continued to make progress. On the positive side, ModMath again 
caused more students (11 percentage points) to enroll in a math class; students offered Mod- 
Math remained much more likely (34 percentage points) to have earned at least one math credit 
since the start of the study; and the program group maintained its advantage with respect to the 
percentage of the developmental course sequence completed (10.9 percentage points). How- 
ever, ModMath had no discernable effect on causing students to achieve the key milestones of 
completing the first half of the developmental math sequence or completing the entire sequence 
and being deemed college-ready in math. Lamentably, only around 15% of the 1,403 students 
in the study became college-ready in math within two semesters. Most students either did not 
attempt all required developmental math courses/modules, failed one or more courses/modules, 
or did not re-enroll at Tarrant County College. 


Third semester. Students were tracked through three semesters after they entered the 
study. Students in ModMath and the more traditional math course had almost identical rates of 
completing the developmental math course sequence (22.5 percent vs. 22.0 percent, respectively). 
This occurred even though program group students continued to attempt a math class (course or 
module) at a higher rate than their control group counterparts, and they maintained their advantage 
with respect to the percentage of the developmental math sequence completed. Nonetheless, when 
it came to three major milestones — completing the first half of the developmental math 
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sequence, becoming college-ready in math, and passing the first college-level math course, there 
was no discernable difference between the outcomes of students offered ModMath and their con- 
trol group counterparts. 


Subgroups Findings 


In addition to examining ModMath’s overall average effects, we explored the programs’ 
effects for different types of students — specifically, with respect to students’ baseline comfort 
with technology, developmental need, intent to enroll full time, and intent to work full time — all 
of which were measured prior to random assignment. Table 8 presents findings by subgroup. For 
example, the first panel shows that ModMath’s estimated effect on becoming college-ready in 
math after three semesters is 1.0 percentage point for students who self-reported being comforta- 
ble with technology at the start of the study and 0.1 percentage points for students who were not. 
The final column in the table, with p value 0.867, shows that these two effect estimates are not 
statistically distinguishable. Stated differently, the 0.9 percentage point difference (1.0-0.1) in 
effect estimates between these two groups could easily have occurred by chance if the program’s 
true effects were the same for both groups. The rest of the table shows that there is not clear 
evidence that ModMath was more effective for any particular subgroups of students — effect 
estimates are mostly near zero, just like the overall average. 


In summary, breaking the developmental math course sequence into six one-credit, com- 
puterized modules led ModMath students to make incremental progress toward completing the 
sequence. Despite this apparent advantage, the ModMath program is no more (or less) effective 
than the traditional developmental math course at helping students complete their developmental 
math requirements. 


Limitations 


This study is one, if not the only, experimental evaluation of a modularized, computer-assisted, 
self-paced developmental math program in a community college setting. As a result, it provides 
an internally valid estimate of the causal effect of such a program on students’ completion of the 
developmental math course sequence. 


One important limitation of the study is that we are unable to assess ModMath’s effect 
on mastering course content, since program and control students did not take a common stand- 
ardized exam or post-test, and it was beyond the scope of the project to ensure they did so. 


Another limitation is that the present study’s findings are specific to ModMath at Tarrant 
County College and may not generalize to other colleges with different institutional contexts, 
different alternative course offerings, or drastically different student populations. Moreover, stu- 
dents could enroll in ModMath only if they joined this study, while students could enroll in other 
course types without restriction. As a result, the findings presented here do not necessarily inform 
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Table 8. Percentages of students ready for college math in the third cumulative semester. 


Percentage of students P-value for 
ready for college math P-value for _ differential 
Sample Program Control Estimated Standard estimated estimated 
Characteristic (%) size group group effects error effects effects 
Comfortable with computers 
at baseline® 0.867 
Comfortable 1,023 22.3 21.3 1.0 2.6 0.709 
Not comfortable 362 23.8 23.7 0.1 4.5 0.982 
N 1,385 
Number of levels placed 
below college math> 0.115 
One level below or higher 221 31.3 33.5 -2.2 6.4 0.729 
Two levels below 190 21.6 29.9 -8.3 6.4 0.194 
Three levels below 541 20.2 14.4 5.7* 3.3 0.084 
N 952 
Intended to enroll full time 
at baseline® 0.950 
Full time 594 24.7 24.0 0.7 3.6 0.845 
Not full time 777 21.4 20.4 1.0 2.9 0.734 
N 1,371 
Intended to work full time 
at baseline® 0.585 
Full time 609 18.3 18.9 -0.6 3.2 0.863 
Not full time 432 26.7 24.4 2.4 4.2 0.579 


N 1,041 


Note. Rounding may cause slight discrepancies in sums and differences. Estimates are adjusted by cohort. A two- 
tailed t-test was applied to differences between research groups. The fall 2016 cohort has data only for part of the 
third semester. A student was defined as ready for college math when the student passed either the final course in 
the developmental math sequence or a college-level math course. Calculations made using data from the baseline 
survey of TCC students, TCC placement test data, and TCC transcript data. 

®Comfortable with technology defined as students who responded as being “extremely comfortable” or 
“comfortable” when asked “how comfortable are you using computers to do school work?” 

Determined by the TSI placement test. One level below or higher includes students who were TSI exempt or had 


the Paaeement waived, previously met TSI requirements, and placed into college math or one level below college 
math. 


“Intended to enroll full time defined as a student who intended to enroll in 12 credit hours or more during the se- 
mester in which he or she was randomly assigned. 


intended to work full time defined as a student who intended to work 30 hours or more per week during the se- 
mester in which he or she was randomly assigned. 

**po = .01. **p=.05. *p =.10. 

tttp = .01. ttp = .05. tp = .10. 
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whether ModMath would be well suited for students with a strong preference for traditionally 
structured lecture classes; such students could experience ModMath differently and therefore 
have different academic outcomes. 


Discussion 


Increasing the academic success of students referred to developmental math courses is a pressing 
priority for community colleges. Modularized, computer-assisted, self-paced remedial math 
courses, such as ModMath, are a widespread reform currently being implemented at institutions 
across the nation; yet, limited causal evidence on the efficacy of this approach exists. The present 
study employed a randomized controlled trial to evaluate ModMath and found that ModMath is 
similarly effective at getting students through the developmental math sequence as other course 
formats, particularly traditional lecture-based courses. 


Sometimes when an evaluation finds that an intervention was no more or less effective 
than the alternative, the lack of impact can be attributed to poor program implementation or a 
weak contrast between the program and control groups. This was not the case for this study — 
for the most part the program’s services were implemented as planned by its designers, and there 
were significant differences between the classroom experiences of ModMath and control group 
students. Given this, we look to the theory of change for insight into the findings. 


One potential explanation for ModMath’s lack of positive impact is that some elements 
of the theory of change that were theorized to lead to improved student outcomes were not real- 
ized in practice. The fine-tuned diagnostic exam did not change placement for the majority of 
ModMath students; the self-paced modules did not result in student acceleration; and the small 
wins gained by earning credits incrementally did not increase completion of the developmental 
math sequence. We discuss each of these, in turn. 


With regard to the diagnostic exam, it is possible that the curricular changes that short- 
ened the sequence from three to two courses just before the start of the study muted the potential 
benefits of the fine-tuned assessment. While this may be the case, many colleges are beginning 
to rely on multiple measures to assess student readiness for college-level work, instead of relying 
on a single exam. Recent studies have shown that this reform may improve students’ outcomes 
in math (Scott-Clayton & Belfield, 2015). For TCC, incorporating multiple measures of assess- 
ment (in addition to a placement exam) may better identify students who would succeed in a 
higher developmental math module or course. 


With regard to self-pacing, we find that more ModMath students slowed down as op- 
posed to accelerated, which is consistent with other research on self-paced computer-assisted 
courses. There is a tension among autonomy, mastery, and acceleration in modularized courses 
(Bickerstaff et al., 2016). Mastery learning requires students to meet certain benchmarks before 
moving forward in the curriculum, which will naturally lead to slower progression. Furthermore, 
since many developmental math students have a history of underperformance in math, and many 
community college students manage school with work and other personal responsibilities, such 
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as childcare, the flexibility offered by a self-paced course makes it more likely that these students 
will progress slower as opposed to faster. While ModMath designers attempted to overcome this 
challenge by issuing pacing calendars to keep students on track, acceleration was not explicitly 
promoted. To promote acceleration in ModMath, and other self-paced courses, while addressing 
the academic and personal needs of developmental math students in community colleges, a triage 
approach may be necessary. Students who are more advanced or have more available time could 
be identified and explicitly encouraged to move faster, through personalized pacing calendars or 
other means. 


Alternatively, developmental math acceleration strategies that tailor and align required 
content to students’ academic interests, as opposed to expecting them to cover more material in 
less time, may be more feasible. These types of strategies are gaining in popularity and some are 
showing early signs of success. For example, a study of the Dana Center’s Mathematics Pathways 
program, which reduces the required number of developmental math courses based on program 
of study, has shown early signs of positive impacts on college readiness (Zachry Rutschow & 
Mayer, forthcoming). Similarly, shifting required course content from algebra to statistics for stu- 
dents in non-STEM (Science, Technology, and Mathematics) majors is gaining in popularity. At 
least one randomized trial, which included co-requisite remediation in the math needed for the 
statistics course, found evidence of improved outcomes (Logue, Watanabe-Rose, & Douglas, 
2016). 


Regarding modularization, we found that ModMath’s modular course structure resulted 
in greater credit accumulation and increased feelings of accomplishment for students, but that 
these small wins were not enough to get students to the finish line. The fact that only about 22% 
of students in either group completed the two-semester developmental math sequence in three 
semesters indicates that far too many students are struggling using either approach. 


Given this, the fact that higher education institutions are searching for ways to get stu- 
dents through the developmental math sequence faster and that early experimental research shows 
more positive results for other types of interventions (i.e., math pathways and mainstreaming), 
we conclude that ModMath is not currently the most promising strategy for developmental math 
reform. It is, however, a solid alternative to traditional lecture-based courses, as it leads to similar 
outcomes. Thus, to accommodate individual student preferences, colleges could offer ModMath 
and expect results similar to those of lecture-based courses. 
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Appendix A 


Estimation Model 


To obtain a regression-adjusted estimate of the causal effect of the opportunity to participate in 
ModMath, we used the following general linear model: 


(1) Y=,BT+yRBte 


where Y is the outcome of interest; T is a treatment assignment indicator, set equal to one if a 
student was assigned to treatment and zero otherwise; RB is a vector of four random assignment 
block indicators (one for each unique cohort in the study); and € is a random error term. Estimated 
effects presented in the tables are the fs for the relevant outcome. Analyses for all academic 
outcomes at all time points include all 1,403 students. 


The probability of being assigned to the treatment group varied across random assign- 
ment blocks;! therefore, inverse probability of treatment weights were used to ensure an unbiased 
impact estimator. Weights were created as follows: 


T.. 1-T.. 
0) m= (E)-10 (2) 
where: 


e 7;; = 1 if individual t in random assignment block j was assigned to the pro- 
gram group and 0 if assigned to the control group. 


e T.; = the proportion of sample members in random assignment block j as- 
signed to the program group (i.e., the average value of T;; in random assign- 
ment block /); and 


e T., = the proportion of all sample members randomly assigned to the program 
group (i.e., the average value of T;; across all sample members). 


'The observed proportions assigned to the treatment group, by chronological cohort, were 52%, 59%, 61%, 
and 60% percent, respectively. 


27 


References 


Ariovich, L., & Walker, S. A. (2014). Assessing course redesign: The case of developmental 
math. Research and Practice in Assessment, 9, 45—57. 


Attewell, P., Lavin, D., Domina, T., & Levey, T. (2006). New evidence on college remediation. 
Journal of Higher Education, 77(5), 886-924. 


Bailey, T., Jenkins, D., & Leinbach, T. (2005). What we know about community college low- 
income and minority student outcomes: Descriptive statistics from National Surveys. New 
York: Community College Research Center. 


Bailey, T., Jeong, D. W., & Cho, S.-W. (2010). Referral, enrollment, and completion in 
developmental education sequences in community colleges. Economics of Education 
Review, 29(2), 255-270. 


Bailey, T., Bashford, J., Boatman, A., Squires, J., Weiss, M., Doyle, W., ... & Wilson, W. 
(2016). Strategies for postsecondary students in developmental education—A practice guide 
for college and university administrators, advisors, and faculty. National Center for 
Education Evaluation and Regional Assistance, What Works Clearing House, Institute of 
Education Sciences. Available at 
http://ies.ed.gov/ncee/wwe/Docs/PracticeGuide/wwc_dev_ed_ 112916. pdf. 


Bickerstaff, S., Fay, M. P., & Trimble, M. J. (2016). Modularization in developmental 
mathematics in two states: Implementation and early outcomes (Working Paper No. 87). 
New York, NY: Community College Research Center. 


Bishop, A. R. (2010). The effect of a math Emporium course redesign in developmental and 
introductory mathematics courses on student achievement and students’ attitudes toward 
mathematics at a two-year college (Unpublished doctoral dissertation). University of 
Southern Mississippi, Hattiesburg, Mississippi. 


Boatman, A. (2012). Evaluating institutional efforts to streamline postsecondary remediation: 
The causal effects of the Tennessee developmental-course redesign initiative on early 
student academic success. New York, NY: National Center for Postsecondary Research. 


Bracco, K. R., Austin, K., Bugler, D., & Finkelstein, N. (2015). Reforming developmental 
education to better support students’ postsecondary success in the Common Core era: 
Core to college evaluation. San Francisco, CA: WestEd. 


Chekour, A. (2017). The effectiveness of computer-assisted math instruction in developmental 
classes. AURCO Journal, 23, 21-30. 


Chen, X. (2016). Remedial coursetaking at U.S. public 2- and 4-year institutions: Scope, 
experiences, and outcomes (NCES 2016-405). U.S. Department of Education. Washington, 
DC: National Center for Education Statistics. 


Dwinell, P. L., & Higbee, J. L. (1989). The Relationship of Affective Variables to Student 
Performance: Research Findings. 


Engle, J. (2007). Postsecondary access and success for first-generation college students. 
American Academic, 3(1), 25-48. 


29 


Epper, R. M., & Baker, E. D. (2009). Technology solutions for developmental math: An 
overview of current and emerging practices. Journal of Developmental Education, 26(2), 
4-23. 


Fay, M. P. (2017). Computer-mediated developmental math courses in Tennessee high schools 
and community colleges: An exploration of the consequences of institutional context 
(Working Paper No. 91). New York, NY: Community College Research Center. 


Gardenhire, A., Diamond, J., Headlam, C., & Weiss, M. J. (2016). At their own pace: Interim 
findings from an evaluation of a computer-assisted, modular approach to developmental 
math. New York, NY: MDRC. 


Ginder, S. A., Kelly-Reid, J. E., & Mann, F. B. (2017). Graduation rates for selected cohorts, 
2008-13, student financial aid, academic year 2015—16; and admissions in postsecondary 
institutions, fall 2016. First look (provisional data) (NCES 2017-084). U.S. Department of 
Education. Washington, DC: National Center for Education Statistics. 


Goldschmid, B., & Goldschmid, M. L. (1973). Modular instruction in higher education: A 
review. Higher Education, 2(1), 15-32. 


Holland, P. (1986). Statistics and causal inference. Journal of the American Statistical 
Association, 81(396), 945-960. doi: 10.2307/2289064 


Kalamkarian, H. S., Raufman, J., & Edgecombe, N. (2015). Statewide developmental education 
reform: Early implementation in Virginia and North Carolina. New York, NY: 
Community College Research Center. 


Logue, A. W., Watanabe-Rose, M., & Douglas, D. (2016). Should students assessed as needing 
remedial mathematics take college-level quantitative courses instead? A randomized 
controlled trial. Educational Evaluation and Policy Analysis, 38(3), 578-598. 


Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2009). Evaluation of evidence- 
based practices in online learning: A meta-analysis and review of online learning studies. 
Washington, DC: U.S. Department of Education. 


Schochet, P. Z. (2008). Technical methods report: Guidelines for multiple testing in impact 
evaluations (NCEE 2008-4018). Washington, DC: National Center for Education 
Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of 
Education. 


Scott-Clayton, J., & Belfield, C. W. (2015). Improving the accuracy of remedial placement. 
New York, NY: Community College Research Center. 


Squires, J., Faulkner, J., & Hite, C. (2009). Do the math: Course redesign’s impact on learning 
and scheduling. Community College Journal of Research and Practice, 33(11), 883-886. 


Subban, P. (2006). Differentiated instruction: A research basis. International Education 
Journal, 7(7), 935-947. 


Taylor, J. M. (2006). The effects of a computerized-algebra program on mathematics 
achievement of college and university freshmen enrolled in a developmental mathematics 
course (Unpublished doctoral dissertation). Texas A&M University, College Station, 
Texas. 


30 


Tinto, V. (1999). Taking retention seriously: Rethinking the first year of college. NACADA 
Journal, 19(2), 5-9. 


Tomlinson, C. A., & Kalbfleisch, M. L. (1998). Teach me, teach my brain: A call for 
differentiated classrooms. Educational Leadership, 56(3), 52-55. 


Twigg, C. A. (2005). Increasing success for underserved students. National Center for 
Academic Transformation. 


Vygotsky, L. (1978). Zone of proximal development. In Michael Cole et al., Mind in society: 
The development of higher psychological processes (pp. 79-91). Cambridge, MA: Harvard 
University Press. 


Weick, K. E. (1984). Small wins: Redefining the scale of social problems. American 
Psychologist, 39(1), 40. 


Xu, D., & Jaggars, S. S. (2011). Online and hybrid course enrollment and performance in 
Washington State community and technical colleges (CCRC Working Paper No. 31). New 
York, NY: Community College Research Center, Columbia University. 


Zachry Rutschow, E., & Mayer, A. K. (forthcoming). Early findings from a national survey of 
developmental education practices. New York, NY: Center for the Analysis of 
Postsecondary Readiness. Manuscript in preparation. 


31 


About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 


Founded in 1974 and located in New York; Oakland, California; Washington, DC; and Los 
Angeles, MDRC is best known for mounting rigorous, large-scale, real-world tests of new and 
existing policies and programs. Its projects are a mix of demonstrations (field tests of promising 
new program approaches) and evaluations of ongoing government and community initiatives. 
MDRC’s staff members bring an unusual combination of research and organizational experi- 
ence to their work, providing expertise on the latest in qualitative and quantitative methods and 
on program design, development, implementation, and management. MDRC seeks to learn not 
just whether a program is effective but also how and why the program’s effects occur. In addi- 
tion, it tries to place each project’s findings in the broader context of related research — in order 
to build knowledge about what works across the social and education policy fields. MDRC’s 
findings, lessons, and best practices are shared with a broad audience in the policy and practi- 
tioner community as well as with the general public and the media. 


Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
as and target populations. Once known primarily for evaluations of state welfare-to-work pro- 
grams, today MDRC is also studying public school reforms, employment programs for ex- 
prisoners, and programs to help low-income students succeed in college. MDRC’s projects are 
organized into five areas: 

e Promoting Family Well-Being and Children’s Development 

e Improving Public Education 

e Raising Academic Achievement and Persistence in College 

e Supporting Low-Wage Workers and Communities 


e Overcoming Barriers to Employment 


Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 


